Skip to content

Improve BibTeX parsing and bibliography rendering - #573

Open
fare wants to merge 36 commits into
racket:masterfrom
fare-patches:doi-dot
Open

fare wants to merge 36 commits into
racket:masterfrom
fare-patches:doi-dot

Conversation

@fare

@fare fare commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Improve BibTeX parsing and bibliography rendering

Improve Scribble's support for rich bibliographies, particularly annotated bibliographies with long, formatted, multi-paragraph notes. The immediate motivation is the bibliography of my book Lambda: the Ultimate Object, but the improvements apply to ordinary academic papers as well.

BibTeX parsing

  • Add a small LaTeX reader supporting common formatting commands, accents, special characters, protected groups and mathematical expressions.
  • Preserve protected groups when parsing author names and other fields.
  • Support LaTeX escapes in URLs and DOIs, generating correct hyperlinks in HTML and LaTeX.
  • Improve parsing of BibTeX string references, including references adjacent to delimiters.
  • Support formatted content and multiple paragraphs in bibliography notes.

The reader deliberately supports a useful subset of LaTeX rather than attempting to implement a complete TeX interpreter. Unknown commands are preserved rather than silently discarded.

Bibliography rendering

  • Replace the old LaTeX \parbox layout, which prevented page breaks within bibliography entries, with needspace, allowing long annotations to span pages. Keep a five-line minimum by default to approximate the previous behavior.
  • Make needspace optional and expose configurable page-breaking and line-breaking settings without imposing new line-breaking parameters on existing documents.
  • Improve numbered bibliography layout using CSS and LaTeX environments, with hanging indentation and aligned citation numbers.
  • Improve formatting across bibliography fields.
  • Add a period after a DOI only when a note follows it, preserving the existing formatting of ordinary entries without notes. Avoid appending punctuation directly to URLs.
  • Improve chapter-number formatting.
  • Add an optional #:accessed field to make-bib, displayed
    immediately after the URL, when one is displayed.
    BibTeX online and webpage entries now use this field
    instead of putting access dates in their location.
    This places access dates after publication dates and
    next to the URLs they qualify, without changing the
    generic ordering of bibliography fields.
    The existing webpage-location API remains unchanged.

The improvements primarily target HTML and LaTeX. Typst receives minimal compatibility support rather than a new rendering implementation.

API, documentation and tests

  • Clarify the distinction between strings, Scribble content and elements in the bibliography implementation. Location helpers may now return content or #f rather than necessarily returning an element. This is a public API change.
  • Update the manuals to document the new behavior and configuration options.
  • Bump scribble-lib to version 1.68.
  • Add unit tests, expected-output tests and rendering tests, including regression tests for accented names, protected groups, escaped hyperlinks and multi-paragraph notes.
  • Add #:accessed to make-bib; access dates are omitted when no URL is displayed, including when a DOI takes precedence.

The changes have been exercised against my book's heavily annotated bibliography and reviewed through several rounds of automated and manual testing.

Implementation was substantially assisted by GPT-5.6 Sol, with additional review by Claude Opus 5.5.

Compatibility notes

  • webpage-location is removed. It was a recent addition from 1.61, I'm not sure anyone uses it directly (as opposed to through bibtex.rkt), but if they do they will have to migrate to #:accessed instead.
  • bibtex-parse now keeps inner braces in raw field values
    (e.g. "The {ACM} Paper") so that protected groups survive until
    rendering; rendered output is unchanged, but code that reads
    bibdb-raw directly will now see the braces.
  • BibTeX titles are no longer passed through verbatim to LaTeX;
    they go through the same reader as other fields. Math and unknown
    commands are still emitted verbatim in LaTeX output.
  • Notes are now decoded like other Scribble text, so -- becomes
    an en dash and straight quotes become curly quotes.
  • A purely numeric #:chapter now renders as "chapter N".
  • In LaTeX output, entries may now break across pages (subject to
    the five-line minimum), and numbered bibliographies use hanging
    indentation instead of a table.
  • In text output, numbered entries are no longer aligned as a table.
  • Various minor bugfixes, in handling inbook, proceedings, or braces in author names.

@fare

fare commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

PS: I'm tempted to have a clanker improve the parsing of bibtex to support (a) emphasis, in notes, (b) {} in titles to prevent lowercasification when using LaTeX.

Also, maybe to improve autobib so multi-paragraph notes are a thing, and maybe crossing page boundaries within a bibliography entry.

These two would also much improve LtUO.

PPS: I'm still thinking about (a), but at least the basics of (b) was apparently done by #571.

PPPS: The clankers did it!

@fare

fare commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Weird: the CI failed on Racket 9.3.0.8, but at home it just worked on v9.3.0.8-2026-09-14-06fd59e02c [cs].

PS: I had forgotten to update a test.

@mflatt

mflatt commented Sep 19, 2026

Copy link
Copy Markdown
Member

The change to add . after a DOI would cause papers written with Scribble+autobib to render differently than Latex+bibtex (in the "acmart" style). Would it work to add . only when there's a note? Or maybe it should be configurable?

@fare

fare commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Yes, adding a "." only if there's a note makes a lot sense, especially for backwards compatibility. Thanks for the suggestion.

PS: also rebasing to the latest master.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 1 file in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 2 files in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 3 files in this pull request and found no issues.

@fare fare changed the title autobib: include "." after doi, to clearly separate from notes. [WIP] bibliography improvements Sep 21, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 3 files in this pull request and found no issues.

Use CSS and LaTeX environments for autobib number-style.
Much nicer output in HTML and PDF. In number-style, text output
won't align as a table anymore, but will have a space after the [number].

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 4 files in this pull request and found no issues.

fare and others added 3 commits September 26, 2026 10:12
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment thread scribble-lib/scriblib/private/read-latex.rkt
Comment thread scribble-lib/scriblib/private/read-latex.rkt Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment thread scribble-lib/scriblib/private/read-latex.rkt Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@fare fare changed the title [WIP] bibliography improvements Bibliography improvements Sep 26, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@mflatt

mflatt commented Sep 26, 2026

Copy link
Copy Markdown
Member

Can you say more about why location helpers need to be able to return #f or content, instead of an element? I see the notes that it changed, but I haven't been able to find the reason it's needed.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@fare fare changed the title Bibliography improvements Improve BibTeX parsing and bibliography rendering Sep 26, 2026
@fare

fare commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

The ability to return #f is so that we can eliminate fields that contain no information, and in turn not have separators between empty fields, which would otherwise happen if there were an opaque element: . , . . , .

Returning content instead of element also seemed more practical to me, considering that the output has a loose structure and is meant to be further aggregated into bigger lists. It seemed to me that at that point, if we really wanted an element, we could better add one in the caller when the content is not #f. But I'm not sure I understand the invariants of Scribble with respect to element vs content vs decoded-content vs blocks vs passes, etc., so I admit I don't really know.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and found no issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resyntax analyzed 7 files in this pull request and has added suggestions.

Comment on lines +262 to +265
(define body
(begin
(read-char ip)
(read-group #t)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

define-begin-extraction: The begin in this definition can be extracted into the surrounding definition context.

Suggested change
(define body
(begin
(read-char ip)
(read-group #t)))
(read-char ip)
(define body (read-group #t))
Debugging details
Textual replacement
(line-replacement
  #:new-lines
    '#("                     (read-char ip)"
       "                     (define body (read-group #t))")
  #:original-lines
    '#("                     (define body"
       "                       (begin"
       "                         (read-char ip)"
       "                         (read-group #t)))")
  #:start-line 262)
Syntactic replacement
(syntax-replacement
  #:introduction-scope #<procedure:...and/syntax-local.rkt:148:2>
  #:new-syntax
    #<syntax:.../resyntax/base.rkt:223:10 (cond ((eqv? (peek-char ip) #\{) (read-char ip) (define body (read-group #t)) (emit! (cond ((string=? word "emph") (apply emph body)) ((string=? word "texttt") (apply tt body)) ((string=? word "textit") (apply italic body)) ((string=? word "textbf") (ap...>
  #:original-syntax
    #<syntax:scribble-lib/scriblib/private/read-latex.rkt:260:18 (cond ((eqv? (peek-char ip) #\{) (define body (begin (read-char ip) (read-group #t))) (emit! (cond ((string=? word "emph") (apply emph body)) ((string=? word "texttt") (apply tt body)) ((string=? word "textit") (apply italic body)) ((string=? word "text...>
  #:source
    (file-source
     #<path:/home/runner/work/scribble/scribble/scribble-lib/scriblib/private/read-latex.rkt>)
  #:uses-universal-tagged-syntax? #f)

@mflatt

mflatt commented Sep 27, 2026

Copy link
Copy Markdown
Member

The ability to return #f is so that we can eliminate fields that contain no information, and in turn not have separators between empty fields, which would otherwise happen if there were an opaque element

I can appreciate why make-bib needs to accept #f for location, but I'm not seeing why a location-generating function like proceedings-location would be called if there was not information about a proceedings to return. What would be an example where that happens?

Returning content instead of element also seemed more practical to me, considering that the output has a loose structure and is meant to be further aggregated into bigger lists.

There shouldn't be a difference between content c and the element (element #f c). If there is, then that's worrying, and I'd like to understand more. If there's not, then I think it may be worthwhile for backward compatibility to add element wrappers on the results — but I'm also not sure it really matters, since content is generally accepted where are element is accepted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants